Recognition of Voice and Hand activities through Fusion of Acceleration and Speech1

Hand activity and speech comprise the most important modalities of human-to-agent interaction. So a multimodal interface can achieve more natural and effective human-agent interaction. In this paper, we suggest a novel technique for improving the performance of accelerometer-based hand activity recognition system using fusion of speech. The speech data is used in our experiment as the complementary sensor data to the acceleration data in an attempt to improve the performance of hand activity recognizer. This recognizer is designed to be capable of classifying nineteen hand activities. It consists of 10 natural gestures, e.g., ‘go left’, ‘over here’ and 9 emotional expressions by hand activity, e.g., ‘I feel hot’, ‘I love you’. To improve performance of hand activity recognition using feature fusion, we propose a modified Time Delay Neural Network (TDNN) architecture with a dedicated fusion layer and a time normalization layer. Our experimental result shows that the performance of this system yields an improvement of about 6.96% compared to the use of accelerometers alone.